安裝 Kubernetes 環境

Kubernetes 環境安裝選擇

Kubernetes的叢集建置方法有很多種，其中最常見的方式有：

Minikube
- 一般使用者可以使用 minikube 在個人電腦建立 K8S 的模擬環境
kubeadm
- 安裝方式最複雜，但也最接近實際使用情境，也是證照考試中必需要使用的
Kubernetes on cloud （ AWS，Azure，GCP 等）
- 在雲端服務建立 K8S cluster 是最容易的，但需要稍微暸解雲端服務是如何設定

Minikube

Minikube 是一個可以快速建立Kubernetes的工具

Minikube Pre-Requirements

是透過建立虛擬機器或 Docker 容器的方式來實現的：

虛擬機，就是創建啟動幾個虛擬機，然後安裝 k8s
透過 docker 建立幾個容器，然後把 k8s 安裝和啟動在這些容器裡

對於 Windows 和 Mac 的機器，建議使用虛擬化的方式進行（也就是說要確保你的機器可以成功的創建和啟動虛擬機器），要滿足以下條件：

BIOS 開啟虛擬化

要有安裝一個虛擬化的 hypervisor
- VirtualBox
- VMware Fusion
- 或其它

Install Steps for Windows

開啟 powershell，運行

New-Item -Path 'c:\' -Name 'minikube' -ItemType Directory -Force
Invoke-WebRequest -OutFile 'c:\minikube\minikube.exe' -Uri 'https://github.com/kubernetes/minikube/releases/latest/download/minikube-windows-amd64.exe' -UseBasicParsing

為了能直接在命令列使用minikube指令，而不是每次都輸入 c:minikubeminikube.exe，有兩種方式：

新增環境變數，以管理員身份開啟 powershell 運行

$oldPath = [Environment]::GetEnvironmentVariable('Path', [EnvironmentVariableTarget]::Machine)
if ($oldPath.Split(';') -inotcontains 'C:\minikube'){      [Environment]::SetEnvironmentVariable('Path', $('{0};C:\\minikube' -f $oldPath), [EnvironmentVariableTarget]::Machine)
}

添加 Alias，以非管理員開啟 powershell 運行

notepad.exe $PROFILE

這指令用 notepad 開啟一個 powershell 的 profile 檔案（如果是第一次運行，那麼會提示你要新建一個文件，點選確定）

把以下內容貼到文件中，儲存關閉文件，然後重新啟動開啟 powershell，就可以直接在命令列中使用 minikube 指令了

Set-Alias -Name minikube -Value C:\minikube\minikube.exePS C:\Users\Peng Xiao> minikube

minikube provisions and manages local Kubernetes clusters optimized for development workflows.

Basic Commands:
start          Starts a local Kubernetes cluster
status         Gets the status of a local Kubernetes cluster
stop           Stops a running local Kubernetes cluster
delete         Deletes a local Kubernetes cluster
dashboard      Access the Kubernetes dashboard running within the minikube cluster
pause          pause Kubernetes
unpause        unpause Kubernetes

Images Commands:
docker-env     Configure environment to use minikube's Docker daemon
podman-env     Configure environment to use minikube's Podman service
cache          Add, delete, or push a local image into minikube
image          Manage images

Configuration and Management Commands:
addons         Enable or disable a minikube addon
config         Modify persistent configuration values
profile        Get or list the current profiles (clusters)
update-context Update kubeconfig in case of an IP or port change

Networking and Connectivity Commands:
service        Returns a URL to connect to a service
tunnel         Connect to LoadBalancer services

Advanced Commands:
mount          Mounts the specified directory into minikube
ssh            Log into the minikube environment (for debugging)
kubectl        Run a kubectl binary matching the cluster version
node           Add, remove, or list additional nodes
cp             Copy the specified file into minikube

Troubleshooting Commands:
ssh-key        Retrieve the ssh identity key path of the specified node
ssh-host       Retrieve the ssh host key of the specified node
ip             Retrieves the IP address of the specified node
logs           Returns logs to debug a local Kubernetes cluster
update-check   Print current and latest version number
version        Print the version of minikube
options        Show a list of global command-line options (applies to all commands).

Other Commands:
completion     Generate command completion for a shell

Use "minikube <command> --help" for more information about a given command.
PS C:\Users\Peng Xiao>

minikube start 以 VirtualBox 驅動程式和 v1.24.0 版的 Kubernetes 為例

minikube start --driver=virtualbox --kubernetes-version=v1.24.0
kubectl

可以透過 minikube 來運行 kubectl

minikube kubectl -- $KUBECTLCOMMAND

為了方便，也可以把下面的 alias 加到 powershell 的 PROFILE 裡

function kubectl { minikube kubectl -- $args }
doskey kubectl=minikube kubectl $

Install Steps for MacOS

x86 晶片

curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-darwin-amd64
sudo install minikube-darwin-amd64 /usr/local/bin/minikube

M1 ARM 晶片

curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-darwin-arm64
sudo install minikube-darwin-arm64 /usr/local/bin/minikube

啟動

x86 晶片

minikube start --driver=virtualbox --kubernetes-version=v1.24.0

x86 晶片

minikube start --driver=docker --alsologtostderr --kubernetes-version=v1.24.0

設定 Alias

alias kubectl="minikube kubectl --"

Kubeadm

如果一切這麼剛好，你懂得使用 Terrafrom，且想要安裝 Kubeadm 在 aws。你可以使用接下來的 Terraform 範例，不懂得使用的話，可以参考官方連結來安裝

環境準備

準備三台 Linux （以 Ubuntu20.04 LTS 為例），三台機器之間能相互通訊

以下是接下來使用的三台 Ubuntu 20.04 LTS

hostname	IP	system	memory
k8s-master	192.168.56.10	Ubuntu 20.04 LTS	4GB
k8s-worker1	192.168.56.11	Ubuntu 20.04 LTS	2GB
k8s-worker2	192.168.56.12	Ubuntu 20.04 LTS	2GB

注意事項

請注意上面準備的機器必須能夠上網，確保機器能存取 Google
如果你使用的是雲端服務提供的虛擬機，請確保把安全策略群組配置好，確保三台機器之間可以存取任意端口，https://kubernetes.io/docs/reference/ports-and-protocols/

安装 containerd、kubeadm、kubelet、kubectl

把下面的 Shell腳本儲存成一個文件（master.sh），放到三台機器裡。

然後分別在三台機器上執行 sudo sh master.sh 運行腳本。

如果要修改 Kubernetes 版本，請修改下面腳本的最後一行，目前我們使用的版本是 1.28.0, 可以透過指令 apt list -a kubeadm 查看可用版本

master.sh

#!/bin/bash
echo "[TASK 1] Disable and turn off SWAP"
sed -i '/swap/d' /etc/fstab
swapoff -a

echo "[TASK 2] Stop and Disable firewall"
systemctl disable --now ufw >/dev/null 2>&1echo "[TASK 3] Enable and Load Kernel modules"
cat >>/etc/modules-load.d/containerd.conf<<EOF
overlay
br_netfilter
EOF
modprobe overlay
modprobe br_netfilter

echo "[TASK 4] Add Kernel settings"
cat >>/etc/sysctl.d/kubernetes.conf<<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
EOF
sysctl --system >/dev/null 2>&1echo "[TASK 5] Install containerd runtime"
mkdir -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo   "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
apt -qq update >/dev/null 2>&1
apt install -qq -y containerd.io >/dev/null 2>&1
containerd config default >/etc/containerd/config.toml
sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
systemctl restart containerd
systemctl enable containerd >/dev/null 2>&1echo "[TASK 6] Add apt repo for kubernetes"
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add - >/dev/null 2>&1
apt-add-repository "deb http://apt.kubernetes.io/ kubernetes-xenial main" >/dev/null 2>&1echo "[TASK 7] Install Kubernetes components (kubeadm, kubelet and kubectl)"
apt install -qq -y kubeadm=1.28.0-00 kubelet=1.28.0-00 kubectl=1.28.0-00 >/dev/null 2>&1

腳本結束以後，可以下 kubeadm，kubelet，kubectl，如果都能獲取到版本，說明安裝成功

kubeadm version

kubelet --version

kubectl version

初始化 master 節點

以下操作都在 master 節點上進行

可以先拉取叢集所需的 images（可選）

sudo kubeadm config images pull

[config/images] Pulled registry.k8s.io/kube-apiserver:v1.28.2
[config/images] Pulled registry.k8s.io/kube-controller-manager:v1.28.2
[config/images] Pulled registry.k8s.io/kube-scheduler:v1.28.2
[config/images] Pulled registry.k8s.io/kube-proxy:v1.28.2
[config/images] Pulled registry.k8s.io/pause:3.9
[config/images] Pulled registry.k8s.io/etcd:3.5.6-0
[config/images] Pulled registry.k8s.io/coredns/coredns:v1.9.3

初始化 Kubeadm

sudo kubeadm init --apiserver-advertise-address=192.168.56.10 --pod-network-cidr=10.244.0.0/16

  Your Kubernetes control-plane has initialized successfully!

  To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

  Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

  You should now deploy a pod network to the cluster.
  Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

  Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.56.10:6443 --token 0pdoeh.wrqchegv3xm3k1ow \
  --discovery-token-ca-cert-hash sha256:f4e693bde148f5c0ff03b66fb24c51f948e295775763e8c5c4e60d24ff57fe82

apiserver-advertise-address 這個位址是本地用於和其他節點通訊的IP位址
pod-network-cidr pod network 位址空間

最後一段的輸出要保存好, 這一段指出後續需要做什麼配置

配置 .kube

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

檢查狀態

$ kubectl get nodes
$ kubectl get pods -A

shell 自動補全 (Bash)

https://kubernetes.io/docs/reference/kubectl/cheatsheet/#kubectl-autocomplete

source <(kubectl completion bash)echo "source <(kubectl completion bash)" >> ~/.bashrc

部署 pod network 方案

在官方連結選擇一個 network 方案，依照提供的具體連結去部署
這次選擇 overlay 的方案，名字叫 flannel 部署方法如下
下載檔案 https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml ，並進行下列修改：確保 network 是我們配置的 –pod-network-cidr 10.244.0.0/16
在 kube-flannel 的容器 args 裡，確保有 iface=enp0s8, 其中 enp0s8 是我們的–apiserver-advertise-address=192.168.56.10 介面名
例如我們的機器，這個IP的介面名稱是 enp0s8

net-conf.json: |  {
    "Network": "10.244.0.0/16",
    "Backend": {
      "Type": "vxlan"
    }
  }

ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
      valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
      valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000    link/ether 02:9a:67:51:1e:b6 brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic enp0s3
      valid_lft 85351sec preferred_lft 85351sec
    inet6 fe80::9a:67ff:fe51:1eb6/64 scope link
      valid_lft forever preferred_lft forever
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000    link/ether 08:00:27:59:c5:26 brd ff:ff:ff:ff:ff:ff
    inet 192.168.56.10/24 brd 192.168.56.255 scope global enp0s8
      valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:fe59:c526/64 scope link
      valid_lft forever preferred_lft forever

把修改好的檔案儲存一個新文件，檔案名稱flannel.yaml，上傳到master節點，然後執行

kubectl apply -f flannel.yaml

檢查結果，如果顯示下面的結果，pod 都是 running 的狀態，表示 network 方案部署成功

vagrant@k8s-master:~$ kubectl get pods -A
NAMESPACE     NAME                                 READY   STATUS    RESTARTS   AGE
kube-system   coredns-6d4b75cb6d-m5vms             1/1     Running   0          3h19m
kube-system   coredns-6d4b75cb6d-mmdrx             1/1     Running   0          3h19m
kube-system   etcd-k8s-master                      1/1     Running   0          3h19m
kube-system   kube-apiserver-k8s-master            1/1     Running   0          3h19m
kube-system   kube-controller-manager-k8s-master   1/1     Running   0          3h19m
kube-system   kube-flannel-ds-blhqr                1/1     Running   0          3h18m
kube-system   kube-proxy-jh4w5                     1/1     Running   0          3h17m
kube-system   kube-scheduler-k8s-master            1/1     Running   0          3h19m

添加 worker 節點

新增 worker 節點非常簡單，直接在 worker 節點上執行 join 即可，注意 –token

sudo kubeadm join 192.168.56.10:6443 --token 0pdoeh.wrqchegv3xm3k1ow \
  --discovery-token-ca-cert-hash sha256:f4e693bde148f5c0ff03b66fb24c51f948e295775763e8c5c4e60d24ff57fe82

找尋 token 和 discovery-token-ca-cert-hash
token 可以透過 kubeadm token list

$ kubeadm token list
TOKEN                     TTL         EXPIRES                USAGES                   DESCRIPTION                                                EXTRA GROUPS
0pdoeh.wrqchegv3xm3k1ow   23h         2022-07-19T20:13:00Z   authentication,signing   The default bootstrap token generated by 'kubeadm init'.   system:bootstrappers:kubeadm:default-node-token

而 discovery-token-ca-cert-hash 可以透過

openssl x509 -in /etc/kubernetes/pki/ca.crt -pubkey -noout |
openssl pkey -pubin -outform DER |
openssl dgst -sha256

結果類似 (stdin)= d301f5ac98d4114cdbe930717705f3bc284243f443c4ff33d32c2cee01bf7945
最後在 master 節點查看 node 和 pod 結果

kubectl get nodes
NAME          STATUS   ROLES           AGE     VERSION
k8s-master    Ready    control-plane   3h26m   v1.28.2
k8s-worker1   Ready    <none>          3h24m   v1.28.2
k8s-worker2   Ready    <none>          3h23m   v1.28.2

pod 的話，應該可以看到三個 flannel，三個 proxy 的 pod

kubectl get pods -A

NAMESPACE     NAME                                 READY   STATUS    RESTARTS   AGE
kube-system   coredns-6d4b75cb6d-m5vms             1/1     Running   0          3h19m
kube-system   coredns-6d4b75cb6d-mmdrx             1/1     Running   0          3h19m
kube-system   etcd-k8s-master                      1/1     Running   0          3h19m
kube-system   kube-apiserver-k8s-master            1/1     Running   0          3h19m
kube-system   kube-controller-manager-k8s-master   1/1     Running   0          3h19m
kube-system   kube-flannel-ds-blhqr                1/1     Running   0          3h18m
kube-system   kube-flannel-ds-lsbg5                1/1     Running   0          3h16m
kube-system   kube-flannel-ds-s7jtf                1/1     Running   0          3h17m
kube-system   kube-proxy-jh4w5                     1/1     Running   0          3h17m
kube-system   kube-proxy-mttvg                     1/1     Running   0          3h19m
kube-system   kube-proxy-v4qxp                     1/1     Running   0          3h16m
kube-system   kube-scheduler-k8s-master            1/1     Running   0          3h19m

Fix node internal IP issue

如果 node 的 internal IP 不對，例如我們想要的 node internal IP 位址是 en0s8 的位址

kubectl get nodes -o wide

NAME          STATUS   ROLES           AGE    VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
k8s-master    Ready    control-plane   42m    v1.28.0   10.0.2.15     <none>        Ubuntu 20.04.4 LTS   5.4.0-113-generic   containerd://1.6.24
k8s-worker1   Ready    <none>          118s   v1.28.0   10.0.2.15     <none>        Ubuntu 20.04.4 LTS   5.4.0-113-generic   containerd://1.6.24
k8s-worker2   Ready    <none>          85s    v1.28.0   10.0.2.15     <none>        Ubuntu 20.04.4 LTS   5.4.0-113-generic   containerd://1.6.24

ip -c a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
      valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
      valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000    link/ether 02:9a:67:51:1e:b6 brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic enp0s3
      valid_lft 72219sec preferred_lft 72219sec
    inet6 fe80::9a:67ff:fe51:1eb6/64 scope link
      valid_lft forever preferred_lft forever
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000    link/ether 08:00:27:e1:e5:69 brd ff:ff:ff:ff:ff:ff
    inet 192.168.56.10/24 brd 192.168.56.255 scope global enp0s8
      valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:fee1:e569/64 scope link
      valid_lft forever preferred_lft forever

修改檔案 /etc/systemd/system/kubelet.service.d/10-kubeadm.conf ，在最後一行結束時增加新的變數KUBELET_EXTRA_ARGS，指定node ip是本機的enp0s8的位址，儲存退出。

/etc/systemd/system/kubelet.service.d/10-kubeadm.conf

# Note: This dropin only works with kubeadm and kubelet v1.11+[Service]Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"

Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"

# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamicallyEnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env

# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.EnvironmentFile=-/etc/default/kubelet

ExecStart=ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS --node-ip=192.168.56.10

重新啟動 kubelet，就會發現本機 master 節點的 internal IP 顯示正確了。

sudo systemctl daemon-reload
sudo systemctl restart kubelet

kubectl get node -o wide

NAME          STATUS   ROLES           AGE     VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
k8s-master    Ready    control-plane   3h55m   v1.26.0   192.168.56.10   <none>        Ubuntu 20.04.4 LTS   5.4.0-113-generic   containerd://1.5.9
k8s-worker1   Ready    worker          3h35m   v1.26.0   10.0.2.15       <none>        Ubuntu 20.04.4 LTS   5.4.0-113-generic   containerd://1.5.9
k8s-worker2   Ready    worker          3h35m   v1.26.0   10.0.2.15       <none>        Ubuntu 20.04.4 LTS   5.4.0-113-generic   containerd://1.5.9

透過同樣的方法可以修改 worker 1 和 worker 2 節點的 internal IP 位址。

Kubeadm 問題排查

官方連結

安裝 Kubernetes 環境

Kubernetes 環境安裝選擇​

Minikube​

Minikube Pre-Requirements​

Install Steps for Windows​

Kubeadm​

環境準備​

注意事項​

安装 containerd、kubeadm、kubelet、kubectl​

初始化 master 節點​

添加 worker 節點​

Fix node internal IP issue​

Kubeadm 問題排查​

Kubernetes 環境安裝選擇

Minikube

Minikube Pre-Requirements

Install Steps for Windows

Kubeadm

環境準備

注意事項

安装 containerd、kubeadm、kubelet、kubectl

初始化 master 節點

添加 worker 節點

Fix node internal IP issue

Kubeadm 問題排查